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ABSTRACT 

Objective: To assess a question-answer pair (QAP) 
database integrated with websites developed for drug 
information centres to answer complex questions 
effectively. 

Design: Descriptive study with comparison of two 
subsequent 6-year periods (1995-2000 and 
2001-2006). 

Setting: The Regional Medicines Information and 
Pharmacovigilance Centres in Norway (RELIS). 
Participants: A randomised sample of QAPs from the 
RELIS database. 

Primary outcome measure: Answer time in days 
compared with Mann-Whitney U test. 
Secondary outcome measure: Number of drugs 
involved (one, two, three or more), complexity 
(judgemental and/or patient-related or not) and 
literature search (none, simple or advanced) compared 
with x 2 tests. 

Results: 842 QAPs (312 from 1995 to 2000 and 530 
from 2001 to 2006) were compared. The fraction of 
judgemental and patient-related questions increased 
(66%— 75% and 54%— 72%, respectively, p<0.01). 
Number of drugs and literature search (>50% 
advanced) was similar in the two periods, but the 
fraction of answers referring to the RELIS 
database increased (13%— 31%, p<0.01). Median 
answer time was reduced from 2 days to 1 (p<0.01), 
although the fraction of complex questions increased 
from the first to the second period. Furthermore, the 
mean number of questions per employee per year 
increased from 66 to 89 from the first to the second 
period. 

Conclusions: The authors conclude that RELIS has 
a potential to efficiently answer complex questions. 
The model is of relevance for organisation of drug 
information centres. 
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ARTICLE SUMMARY 



Article focus 

■ A model with a national network of drug 
information centres in Norway (RELIS) with 
a common QAP database integrated with 
websites has been developed. 

■ This article describes how the model makes it 
possible to effectively answer complex 
questions. 

Key messages 

■ A randomised sample of QAPs from two 
subsequent 6-year periods after RELIS started 
in 1995 were analysed retrospectively. 

■ Comparison of the second to the first period 
shows RELIS' potential to answer complex 
questions with reduced answer time and 
more frequent use of previous answers in the 
database. 

Strengths and limitations of this study 

■ A randomised sample from the two periods was 
analysed. 

■ The study is descriptive, and analysis is 
retrospective with poorly defined variables. 

■ Development of RELIS (personnel and tech- 
nology) as well as the internet during the total 
study period suggests that interpretation of the 
results should be performed with caution. 



INTRODUCTION 

The regional medicines information and 
pharmacovigilance centres (RELIS) were 
established in 1995 with two regional centres. 
Two additional centres were started in 1997 
and 1998 with a fifth added in 2001. Today, 
RELIS is a national network made up of four 
(two centres in Oslo became one in 2010) 
drug information centres (DICs). They are 
localised at regional university hospitals in 
Norway, where pharmacists and clinical 
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pharmacologists answer questions from healthcare 
professionals. In 1995, RELIS started developing a full- 
text question— answer pair (QAP) database through 
cooperation with a software company (Arnett AS, 
Bergen, Norway) . The database was originally an internal 
(not public) Microsoft Access application but was 
converted to a web-based SQL database with separate 
public (through the RELIS homepage) and internal 
(through the RELIS intranet) interfaces in 2000—2001. 
QAPs are indexed and published by the staff to be 
searchable in Norwegian for healthcare professionals 
free of charge through the public interface. QAPs 
unsuitable for the public domain (eg, questions 
answered solely by a copy of a previous answer, questions 
which lack public interest and not associated with drugs 
or questions that contain information that could identify 
a patient through description of a rare disease) are 
allocated to the internal interface (RELIS internal 
database) only. 

QAPs have been retrieved from the database through 
a simple or advanced search function since 1995. Today, 
the RELIS public database by default contains a simple 
search function where a drug or text word (eg, hyper- 
tension) is entered, but a more advanced search func- 
tion with Boolean operators (AND/OR/NOT), 
categories (eg, interactions, pregnancy), text word, 
individual centre with question number is an option. In 
the internal interface, the advanced search function is 
default. The result of a search (eg, loratadine AND 
pregnancy) is presented as a report of QAPs organised 
chronologically (the newest on top). An individual QAP 
in the report could contain a previous answer (in the 
RELIS public database) as a linked reference available in 
full text. 

The websites include the RELIS homepage (http:/ / 
www.relis.no) targeted at healthcare professionals and 
the RELIS intranet available for RELIS only. The 
websites were developed in 2000, updated in 2010 and 
represent the public and internal domain of the RELIS 
network, respectively. On the homepage, RELIS publish 
relevant drug information including relevant QAPs from 
the database, announce meetings, present recent publi- 
cations and describe news from the pharmacovigilance 
work. Importantly, healthcare professionals can register 
for a monthly newsletter (email) with selected drug 
information from the homepage. The intranet functions 
as a medium for all activity and cooperation between the 
centres in Norway. Thus, it contains important docu- 
ments, training and quality guidelines, discussions and 
information to all employees. In addition it includes 
dedicated sections for individual centres, common 
working groups and projects. 

In 1995 and up to 2006, the majority of questions to 
RELIS were communicated through telephone followed 
by email. However, in the last years, questions have been 
mainly submitted through a web question form available 
on the homepage. Healthcare professionals register 
demographic data (occupation and working address), 



the question (the patient being made anonymous) and 
a deadline for the answer (answer time). Demographic 
data are used for distribution of questions to the 
respective centres, but all information in the web ques- 
tion form are shared by all RELIS staff, regardless of 
location, through exclusive emails. Other questions 
(through telephone, fax, letters or ordinary email) not 
submitted through the web-based question form have 
still been shared by all staff since 1995 through an 
'answer in progress' functionality in the RELIS internal 
database. The name of the employee who received the 
question and the one handling it is also registered. In 
the case of particularly complex questions involving 
several authors, co-signatures are used. In this way, the 
handling of a question is fully traceable and answers to 
subsequent similar questions can be based on re-use of 
QAPs and communication with the authors. 

Physicians consider the information provided by 
RELIS in general and on drug use in pregnancy in 
particular to be of high quality and of significant clinical 
impact. — 8 Questions to the centres are mainly about 
psychoactive drugs and adverse effects similar to other 
DICs. 4 5 Less is known about complexity of questions 
and how this influences answer time. In the present 
study, we retrospectively analysed a randomised sample 
of QAPs in two time periods after RELIS started. The 
objective was to examine if this model for DICs has 
a potential to answer complex questions effectively. 

METHODS 

Contents of the RELIS database were analysed to illus- 
trate how questions were answered. Two periods of 
6 years were compared (1995-2000 and 2001-2006). 
QAPs published in the public database in the respective 
periods were used. They reflect the process of answering 
better than QAPs unsuited for publication as described 
above. Overall publication frequency in the period 
(1995-2006) was 80% (range 66%-95% between indi- 
vidual centres). Relevant data from QAPs, including 
question number, were transferred to SPSS V.17.0 (SPSS 
Inc.) for analysis and subjected to a randomisation 
procedure in the application. From a total of 9697 QAPs, 
a randomised sample of 963 QAPs was used for further 
analysis. Each QAP was compared with the information 
in the associated paper version (all documents and 
information on answering the question) saved in the 
respective RELIS centre. The paper version contained 
the day the question was received and the corresponding 
day an answer was provided. One hundred and twenty- 
one QAPs were excluded because the questions 
concerned nutritional or herbal medicine (n=65), non- 
medical products from pharmacies, homeopathy, 
medical equipment, disinfection or chemicals (n=26), 
or questions that were answered solely by a previous 
answer or by sending copies of relevant scientific articles 
(n=30). Exclusion of these QAPs was based on the 
notion that DICs primary function and resources 
(personnel and available drug information sources) are 
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directed to answer questions concerning drugs. Thus, 
based on a sequence of selection criteria (publication or 
not, randomisation to 10% and the final exclusion), 842 
QAPs were included in the study. They were analysed for 
the number of drugs involved in the question, 
complexity, the type of literature search performed, use 
of references (the number of references used and/or 
use of the RELIS database as a reference) and the time 
needed for providing an answer (in days). The number 
of drugs involved in the queries were categorised as one, 
two or three or more. Questions concerning groups of 
drugs, for example, antipsychotics, were categorised as 
three or more drugs. Judgemental and/ or patient- 
related questions were defined as complex. Factual 
questions, such as the therapeutic dose of a drug or its 
half-life, can easily be located in textbooks, monographs 
or databases. Judgemental questions require by defini- 
tion the integration of data or knowledge and experi- 
ence in the process of making a decision regarding 
a specific therapeutic problem. 6 The original definition 
assumed that answers to judgemental questions could 
not be given in any single reference source, but this 
assumption has been excluded in the present and other 
studies. 7 Judgemental questions are frequently patient- 
related because they involve drug information applied to 



a clinical situation associated with patient-specific char- 
acteristics. They can, however, also represent a more 
general drug-related problem. If the question was 
answered without searching the literature and without 
consulting colleagues, the literature search was cate- 
gorised as none. If it was necessary to search the RELIS 
database, databases containing monographs like the 
Micromedex, the Summary of Product Characteristics 
for the drug, reference books and/ or colleagues/ other 
health professionals only, the search was categorised as 
simple. If searches in databases like Medline, Embase or 
Cochrane to obtain original articles were necessary, the 
search was categorised as advanced. The number of 
questions received each year were analysed by use of the 
statistical function in the database. This function allows 
RELIS to quantify the activity in the centres (eg, the 
number of questions from physicians or pharmacists 
from a particular county from a particular time period to 
a particular RELIS can be retrieved). The number of 
employees in the centres in the respective periods was 
estimated but did not include all cases of maternity 
leave, leave of absence and vacant positions. The mean 
number of questions per employee per year was calcu- 
lated from data corresponding to the start (year) and 
end (year) of each period. Figure 1 summarises the 
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Figure 1 Healthcare professionals with questions can search the Regional Medicines Information and Pharmacovigilance 
Centres (RELIS) public database or submit a web question form through the RELIS homepage. The questions with deadline and 
demographics are distributed to all RELIS staff, who have access to the question— answer pairs (QAPs) throughout their 
processing in the RELIS internal domain. All QAPs are stored in the RELIS internal database and selected for publication in the 
RELIS public database. A randomised sample of QAPs from 1995—2000 to 2001—2006 were retrospectively analysed and 
compared with regard to number of drugs involved (one, two, three or more), complexity (judgemental and/or patient-related or 
not), literature search (none, simple or advanced) and answer time (in days). 
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present RELIS model together with the study sample 
and design. 

Statistics 

The data were analysed using SPSS V.17.0 (SPSS Inc). 
Proportions between groups were analysed using % 2 
comparisons. Discrete data between groups were 
analysed using Mann— Whitney U test, p Values <0.05 
were considered significant. 

RESULTS 

Table 1 shows characteristics of a randomised sample of 
842 QAPs from 1995 to 2000 (n=312) and 2001-2006 
(n=530), respectively, from the RELIS public database. 
The mean number of questions per employee per year 
increased from 66 to 89 from the first to the second 
period. 

DISCUSSION 

This paper describes a QAP database integrated with 
websites for RELIS in Norway. The model is charac- 
terised by a traceable answering process of questions 
submitted to the centres, which are stored in full text in 
a searchable database. Continuous growth of the data- 
base makes it a valuable source of drug information for 
healthcare professionals. Our analysis of two periods 
after RELIS started demonstrates the potential of the 
database to be an effective reference source through re- 
use of previous answers. Furthermore, RELIS represents 
a model for DICs where the staff efficiently answers 
complex questions through cooperation in a network. 

The RELIS model includes quality standards and in- 
house training which ensures that the process of 
answering questions is similar across the centres. The 
intranet contains several guidelines on how to handle 
QAPs (eg, how to write references, how to index in the 
database). Furthermore, each centre uses the same 



commercial drug information databases and textbooks as 
sources. How to answer a question is discussed within 
each RELIS by informal discussions or use of co-signa- 
ture. It is also discussed between different centres due to 
the transparency and traceability of the model and 
regularly in the annual national meetings where all 
centres participate. Thus, a person trained and located in 
one centre can do work for a different centre through 
this model in case of shortage of staff or increased 
workload in one of the regions. In this way, the individual 
centres are effective backups for each other. The 'answer 
in progress' function facilitates communication about 
new questions. Similar questions submitted simulta- 
neously or subsequently can be answered based on 
previous work and communication between individual 
centres. In this way, unnecessary double work can be 
reduced and complex questions asked frequently can be 
answered within a short time frame. 

Our data show that the re-use of the RELIS database as 
a reference source in the answering of new questions was 
significandy higher in 2001-2006 than in 1995-2000. 
This is not surprising, as the number of available QAPs in 
the second time period was higher and the chance of 
finding relevant information increases with the number 
of questions received at RELIS. Re-use of the database as 
a source of information is expected to decrease the 
workload of repeated questions, but patient-specific 
characteristics of relevance to the answer process still has 
to be taken into account. Furthermore, complex ques- 
tions not answered previously are associated with 
increased workload. This is illustrated by the finding that 
the average number of references in the QAPs did not 
decrease from the first to the second period and 
advanced search strategies were still used in most cases. 
We recently performed a prospective study where the 
extent of literature search was the individual factor best 
predicting the time spent answering questions in RELIS. 8 



■ 



Table 1 Characteristics of a randomised sample of 842 question— answer pairs in the RELIS public database 1995—2006 



Period 


1995-2000 


2001-2006 


p Value 


Questions, n 


312 


530 




Drugs involved, category: % 


One: 49 


One: 45 


0.434§ 




Two: 19 

Three or more: 32 


Two: 22 

Three or more: 34* 




Judgemental questions, n (%) 


207 (66) 


399 (75) 


0.005§ 


Patient-related questions, n (%) 


170 (54) 


379 (72) 


<0.001§ 


Answers, n 


312 


530 




Literature search, type: %t 


None: 3 


None: 2 


0.315§ 




Simple: 43 


Simple: 40 






Advanced: 54 


Advanced: 58 




Reference to RELIS database, n (%) 


39 (13) 


164 (31) 


<0.001§ 


References, median (average) 


3(4) 


4(5) 


0.100H 


Days, median (average)^ 


2 (8.4) 


1 (5.0) 


o.oon 



The total number exceeds 1 00% because of rounding off to the nearest whole number. 

fSimple literature search included monographs (eg, Martindale, Summary of Product Characteristics), advanced literature 
articles (eg, databases like Medline and Embase). 

^Answer time is influenced by some extreme outliers (>100 days) and choice of deadline for answer ('not important' is an 
§X 2 Test for contingency tables. 
UMann— Whitney U test for two independent samples. 



search included 
option). 
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However, in RELIS, repeated questions are subjected to 
a new literature search if 3 months have passed since 
a previous answer was provided, and this was more likely 
to happen in the period 1995—2006. With repeated 
questions today, a previous answer in the database is 
more likely to be updated and could be re-used. An 
illustrating example is the common question of antide- 
pressants and pregnancy, which is usually patient-related 
and judgemental in its form. This type of question is 
frequently submitted to RELIS because information in 
product monographs is categorical and not useful for 
solving a clinical situation with a depressed pregnant 
patient in need of drug therapy. A recent answer from 
the database with updated literature search can be 
provided within much shorter time limits than would be 
expected. 

Interestingly, the answer time has been reported to be 
the parameter associated with dissatisfaction with DIGs 
among physicians. A DIG in Denmark reported a median 
time to answer (final answer in contrast to preliminary 
answer) of 25 days among satisfied physicians and 56 days 
among unsatisfied physicians. 9 In our material, the 
median time before a written answer was submitted to 
the questioner was 2 days and 1 day in the periods 
1995-2000 and 2001-2006, respectively. In the 
subgroup of judgemental and patient-related QAPs, the 
corresponding reduction was 4 to 2 days, and 2 to 1 day, 
respectively (data not shown). The average number of 
days before a written answer was given was reduced from 
8 to 5 days in the two periods. These figures are influ- 
enced by a few extreme outliers (>100days before an 
answer were documented as given). Furthermore, 'not 
important' is an option when healthcare professionals 
specify their deadline for the answer. Some questions are 
dependent on documentation from external sources (eg, 
pharmaceutical industry) not controlled by RELIS. Thus, 
the median number of days before an answer is given is 
probably more representative for the service than the 
average. However, the answer time depends greatly on 
how fast the questioner needs his answer and how many 
other questions that are received in the same time 
period. Increased staffing could also influence answer 
time. However, our calculation of the mean number of 
questions per employee per year does not account for all 
the increase in activity during the period after RELIS 
started. Several questions answered by telephone were 
not documented in the database, and in 2003, the 
centres increased the number of employees mainly to 
handle adverse drug reaction reports due to new tasks 
in pharmacovigilance. In the prospective study, we 
measured the time spent to answer each of 96 questions. 8 
The mean time needed to provide an answer was 4.0 h, 
whereas the median time was 3.0 h (range 0.5—24 work 
hours). This represents effective time, not the number of 
workdays (including other work and lunch) recorded in 
the present material. Interestingly, in the same study we 
also observed a minor impact of the number of drugs on 
answer time. This could be due to the experience that 



questions about drug interactions are managed effec- 
tively by use of drug interaction software. 8 

The number of patient-related questions was signifi- 
cantly higher in the second time period than in the first 
one, as were the number of judgemental questions. 
These two variables are related to each other in such 
a way that patient-related questions often are associated 
with judgemental considerations and vice versa. In the 
present study, they were used to indicate complex 
questions. In an earlier study, we found that more than 
60% of the queries were judgemental and most of the 
queries were patient-related. 8 The sample in that study 
was much smaller compared with the present results, but 
the numbers corresponds well. However, the assessment 
of judgemental/non-judgemental is quite subjective, 
and others have used the term consultative for questions 
that requires clinical advice on a special case. Consulta- 
tive questions usually entail discussion with the inquirer 
on possible benefits and hazards of one or more courses 
of action in a clinical case. 10 The justification for using 
judgemental classification versus consultative and other 
classifications is not clear-cut, and this makes compari- 
sons between different studies difficult. However, DICs 
in Norway are similar to the Swedish, which have 
reported a particularly high proportion of consultative 
queries. 11 During the history of RELIS, the number of 
questions from physicians has steadily increased in 
proportion to other groups of healthcare professionals. 
In 2011, 65% of 2565 questions came from physicians 
(40% of physicians from general practice), while 84% 
were patient-related. 

Adverse effects and pregnancy represent the two most 
common categories of questions to RELIS and are 
a substantial part of the QAP database. Feedback to 
healthcare professionals who submit adverse drug reac- 
tion reports commonly involve use of this database. 
Furthermore, several reports were originally generated by 
a question by a physician concerning possible adverse 
effects. Thus, the database and the websites are strongly 
integrated with the pharmacovigilance work. Our answers 
have been evaluated to be of high quality and of signifi- 
cant clinical impact in several studies. x ~ 3 These studies 
cover the period of 1995—1998 and 2006. Interestingly, 
the study from 2006 concerned drug information in 
pregnancy, which represents a category of questions 
where re-use of information in the database is particularly 
frequent. 3 However, such evaluations are performed by 
a selected group of physicians and cannot be generalised 
to other healthcare professionals. Although conse- 
quences on a patient level are usually not known in these 
results, a particularly relevant finding was that almost 10% 
of physicians who evaluated the service reported that the 
information prevented unnecessary termination of 
a pregnancy. 3 

The present model could be of international interest 
for existing or planned network of DICs. A recent survey 
of 75 DICs in the USA found an increase in the number 
of complex questions in 70%, while 53% reported an 
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increase in the time required to answer each question. 
A shared database represents a useful tool for uniform 
handling and documentation of complex questions. 
Furthermore, it becomes an increasingly important drug 
information source for healthcare professionals as well 
as employees in the network. Through a solution with 
web-based communication, the model could be of 
particular importance in poor and undeveloped coun- 
tries and continents with frequent epidemics, outbreaks 
and health problems where rapid dissemination of drug 
information is essential. 

Conclusions 

We conclude that RELIS represents a model with 
a potential to efficiently answer complex questions. The 
model is of relevance for organisation of DICs. 
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